Papers with diagnostic classifiers
Lost in Space: Probing Fine-grained Spatial Understanding in Vision and Language Resamplers (2024.naacl-short)
Copied to clipboard
| Challenge: | Recent approaches for developing vision and language models leverage existing vision and a language expert and try to learn a mapping between them. |
| Approach: | They propose to use a resampler module to create a ‘visual prompt’ which is then fed to the large language models (LLM) using a textual prompt. |
| Outcome: | The proposed method has been shown to be effective across coarse-grained tasks like image captioning and visual question answering, but more fine-grounded tasks that require spatial understanding have not been thoroughly examined. |
Analyzing analytical methods: The case of phonology in neural models of spoken language (2020.acl-main)
Copied to clipboard
| Challenge: | Recent studies have focused on the strengths and weaknesses of various methods for analyzing phonology representations. |
| Approach: | They propose to use diagnostic classifiers and representational similarity analysis to quantify to what extent phonemes and phoneme sequences are encoded. |
| Outcome: | The proposed method is based on two commonly applied techniques . it shows that global-scope methods yield more consistent and interpretable results . |
Internal and external pressures on language emergence: least effort, object constancy and frequency (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies show that the emergent languages rarely display salient features inherent to natural languages, such as compositionality of meaning and generalisation to novel objects. |
| Approach: | They propose to formalise the principle of least effort through an auxiliary objective and explore several game variants inspired by the principle 'object constancy' they find that the proposed sources of pressure result in emerging languages with less redundancy, more focus on high-level conceptual information, and better abilities of generalisation. |
| Outcome: | The proposed sources of pressure result in emerging languages with less redundancy, more focus on high-level conceptual information, and better abilities of generalisation. |
Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Models (2020.acl-main)
Copied to clipboard
| Challenge: | LSTMs can capture syntactic rules in artificial languages, but it is unclear whether they are as capable in natural languages. |
| Approach: | They propose a causal account of structural properties as carried by paths across gates and neurons of a recurrent neural network that localizes and segments the concept into a set of gate or neuron-level paths. |
| Outcome: | The proposed model improves on a widely-studied multi-layer LSTM language model showing that it can learn subject-verb number agreement in English. |
Language Models Use Monotonicity to Assess NPI Licensing (2021.findings-acl)
Copied to clipboard
| Challenge: | Neural language models (LMs) have become powerful approximators of human language . fewer studies have been done on what kind of formal semantic features are encoded by LMs . |
| Approach: | They propose a series of experiments that investigate the semantic knowledge of language models . they use diagnostic classifiers, linguistic acceptability tasks and a ranking method to investigate the models' inner workings. |
| Outcome: | The proposed method can be applied to LMs trained on filtered corpora and gain stronger insights into their generalizations. |
Probing for Constituency Structure in Neural Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Using standard probing techniques, we examine whether contextual neural language models implicitly learn syntactic structure. |
| Approach: | They investigate to which extent contextual neural language models implicitly learn syntactic structure. |
| Outcome: | The proposed model is able to represent constituents of different categories within the neuron activations of a LM such as RoBERTa with high performance even on manipulated data. |
CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning (2020.acl-main)
Copied to clipboard
Alessandro Suglia, Ioannis Konstas, Andrea Vanzo, Emanuele Bastianelli, Desmond Elliott, Stella Frank, Oliver Lemon
| Challenge: | Approaches to Grounded Language Learning focus on a single task-based final performance measure which may not depend on desirable properties of the learned hidden representations. |
| Approach: | They propose an evaluation framework for Grounded Language Learning with Attributes based on three sub-tasks: 1) Goal-oriented evaluation; 2) Object attribute prediction evaluation; and 3) Zero-shot evaluation. |
| Outcome: | The proposed framework evaluates the quality of learned representations with respect to attribute grounding. |